Africa Environmental Data Pipeline

Overview

The data processing targets contained in the predictor_data_processing_targets.R file download and process environmental and data for epidemiological modeling and forecasting across the African continent.

Data Sources

Static Data (Time-Invariant)

Dynamic Data (Time-Varying)

Processing Steps

  1. Data Download: Fetches data from original sources or AWS cache
  2. Preprocessing: Standardizes all data to 0.1° spatial resolution
  3. Temporal Processing:
    • Interpolates satellite data to daily intervals
    • Calculates historical means for each day-of-year
  4. Anomaly Calculation: Computes deviations from historical baselines
  5. Forecast Processing: Creates anomalies for different lead times (0-30, 30-60, 60-90 days)
  6. Integration: Joins all data sources into unified parquet files

Infrastructure Features

Output

The pipeline produces a comprehensive dataset with: